Repository navigation
fix(agent): scope the autocast fallback to the failed request - #459
NandhaKishorM merged 1 commit into
Conversation
|
Thank you for picking up this follow-up from #351, a single bad op should not slow the process down for good. Two things before it can merge: tests/test_runtime_fixes.py still asserts the old behaviour ('infer/falls back and disables amp') and now fails, so please update it to the new contract. Also, since an unsupported op usually fails the same way every time, please add the consecutive-failure threshold the issue mentioned (for example, disable AMP with a warning after 3 failures in a row), so a build that lacks an op does not pay for two forward passes on every request. #349 has landed, so please rebase; the only overlap is in tests/test_portability.py. |
rahul05ranjan
left a comment
There was a problem hiding this comment.
LGTM — correct fix for a real state-leak bug, with thorough tests.
The bug is real: when an MPS/CPU build hit an unsupported autocast op, the fallback did self.amp_enabled = False; self.dtype = torch.float32 and retried — but those mutations stuck for the life of the process, so one unsupported op silently slowed every later request (#351).
The fix changes run() to accept an enabled override and retries the single failed request with run(enabled=False), leaving the agent's AMP settings untouched. The enabled=None default preserves the existing recompute-on-retry behavior for the OOM/CPU-fallback path.
The tests are strong:
autocast/answers after an unsupported opverifies the retry succeeds and AMP stays enabled.autocast/a later request still answers+autocast/later request is a single forwardprove the fix is scoped (no lingering slowdown).autocast/repeated failure leaves amp enabledcovers the case where the retry also fails, confirming AMP isn't left off.
Minor nit (non-blocking): the run(enabled=None) signature uses None as a sentinel for "recompute", which is slightly subtle — a comment already explains it, but an explicit keyword like run(recompute=True) or a separate helper would be marginally clearer. Not worth blocking on.
This is a clean, well-scoped fix.
9e17ba2 to
0e737ce
Compare
|
Rebased onto main after #349. |
Agent exposes cpu_fallback_count and last_fallback_reason, recorded under the OOM fallback lock, and GET /health reports them per loaded agent so a silent GPU->CPU degradation is visible to operators. Part 1 of NandhaKishorM#351 (observability); NandhaKishorM#459 covers part 2 (scoped autocast drop).
0e737ce to
b775efc
Compare
|
Rebased onto current main. The autocast behavior is the same: one miss retries in full precision and leaves AMP on, and AMP turns off after three misses in a row. |
Retry an unsupported autocast op in full precision. Turn AMP off only after three failures in a row, so a build that lacks the op does not pay for two forwards on every later request.
|
Passed checks and merged. Thank you @MoonPointer-Byte! It ships in 0.3.22. |
What
Agent._inferretries an unsupported autocast op in full precision. AMP stays on until three such failures in a row, then it is turned off and a warning is printed. A forward that succeeds under autocast clears that streak.Why
On MPS or CPU, one unsupported op used to set
amp_enabled = Falseanddtype = float32for the rest of the process. Leaving AMP on forever means a build that lacks the op pays for two forwards on every request. This is the autocast half of #351. Rebased onto current main; the CUDA OOM path, including the/healthfallback counters, is unchanged.How it was verified
test_runtime_fixes.py: 51 passed.test_portability.py: 35 passed.Follow-ups
None on this change.